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Purpose 

The purpose of this paper is to inform evaluators of a method of evaluating capacity building in 
educational organizations or programs. Here the method is presented in the context of an 
evaluation of the U. S. Department of Education's North Central Comprehensive Center (NCCC). 

Perspective 

One definition of capacity building is helping an organization increase its ability to fulfill its 
mission (Wing, 2004). In the nonprofit sector, funders (such as foundations) that focus on 
capacity building aim to increase the capacity of the nonprofit organizations that they support 
so that these organizations are high-performing and effective (McKinsey and Company, 2001). 
The same principles apply to programs that aim to increase the capacity of educational 
organizations, such as state education agencies (SEAs). Evaluators of capacity-building efforts 
therefore must focus on evidence that the program has helped to build the capacity of the 
organization it serves. 

The 16 regional U. S. Department of Education Comprehensive Centers (CCs) each serve a 
specific geographic area. The five-year mission of the CCs is to provide technical assistance that 
builds the capacity of state education agencies to help districts and schools meet their student 
achievement goals and the requirements of NCLB. The NCCC, serving the states of Iowa, 
Minnesota, Nebraska, North Dakota, and South Dakota, has adopted the following objectives 
provided by the U.S. Department of Education: 

1. Increase State capacity to assess the improvement needs of districts and 
schools. 

2. Increase State capacity to develop solutions to address improvement needs 
of districts and schools. 

3. Increase State capacity to build and sustain systemic support for district and 
school improvement efforts related to closing achievement gaps and 
improving student outcomes 

4. Increase State capacity to improve the tools and systems for school 
improvement and accountability. 

State agency staff work collaboratively with CC staff to identify projects and a plan for 
collaborative action, under one or more of these objectives. 

The performance of the NCCC is measured both by its ability to provide timely, high-quality 
technical assistance that addresses the needs of the region and by the extent to which that 
assistance is successful in building the capacity of participating SEAs. While evaluators are 
usually accustomed to measuring the timeliness and quality of delivered services, some may 
not have previous experience in evaluating capacity building. Evaluating capacity can be a 
challenge because of the dynamic and multi-dimensional nature of capacity within any 
organization, and because "capacity" is a difficult construct to measure. In this case, evaluators 
created a logic model (see Figure 1) in order to better explain the capacity-building approach of 
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the NCCC. It is guided by previous work in best practices in capacity building evaluation, such as 
Connolly and York (2002) and Wing (2004). The logic model delineates how the technical 
assistance efforts of NCCC staff can impact capacity in terms of both the immediate outputs, as 
well as the short, intermediate, and long term impacts. The overarching concept of the 
capacity building logic model is that capacity is built in steps: first the attendees participate in 
activities that they find useful, relevant, and of high quality; then they build new knowledge and 
gain self-efficacy (perceived confidence) that they can improve services to districts and schools; 
and finally they apply their knowledge to deliver improved services. 

Methods 

In this evaluation, both quantitative and qualitative data are used to provide the most complete 
formative and summative information about the results of capacity building activities. The data 
sources are described below in terms of the outputs and outcomes in the logic model. 

Although each technical assistance activity has its own specific goal(s) and objective(s), the 
overriding objective of all the technical assistance is to support state education agencies in 
becoming more capable of assisting districts and schools in need of improvement. While this 
ultimate question cannot be fully answered until the end of the five-year award period, ongoing 
data collection, by project, helps illuminate progress towards this objective. To this end, 
evaluators examined the resources used to enact the program (inputs), such as funding, NCCC 
staff time and expertise. 

To provide formative evaluation on the outputs, evaluators focused on the question: "How is 
the NCCC technical assistance being provided?" Data sources include artifacts such as 
participant lists, meeting agendas, and activity reports. 

In evaluating capacity building outcomes, evaluators considered the short, intermediate, and 
long-term impacts of NCCC technical assistance. Data sources included surveys, interviews, and 
focus groups, as well as any products (e.g., action items, tools) developed by the attendees as a 
result of the technical assistance. 

The first step in the outcomes area of the logic model was to collect data on the short term 
outcomes and impact of participation in NCCC technical assistance activities. These data answer 
two primary formative evaluation questions: 

1. How do the participants react to the technical assistance activities? (What do they 
like and dislike about them?) 

2. To what extent are the technical assistance activities found to be useful, 
relevant, and of high quality? 

To address these questions, evaluators used surveys to determine participants' perceptions of 
the quality, relevance and utility of the technical assistance; if participants did not find the 
activities to be valuable, it is unlikely that they would go on to achieve the other planned 
outcomes. 
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The second step in the outcomes area of the logic model is to determine whether activities 
helped participants gain useful knowledge and self-efficacy (perception of competence) for 
assisting districts and schools— important precursors to useful action. The relevant summative 
evaluation questions are: 

1. What did participants learn as a result of the technical assistance activities? 

2. To what extent do they perceive themselves as more capable of assisting districts 
and schools? 

To address these questions evaluators use survey items that ask about knowledge and 
perceived capability, and follow up with interviews and focus groups. 

The final step in the outcomes area of the logic model is to collect data on the long term impact 
of participation in NCCC technical assistance activities, as participants apply the skills developed 
through technical assistance activities to their areas of need. These data answer two summative 
evaluation questions: 

1. How did the participants apply the knowledge from technical assistance 
activities in their services to districts and schools? What have they done 
differently? 

2. What evidence is there that services to districts and schools have improved? 

Data sources include follow-up surveys and interviews of participants, as well as participant- 
developed documents and artifacts. 

Data Sources 

Surveys 

Immediate feedback about events is collected from participants via surveys in order to assess 
the short-term outcomes of perceived quality, relevance, and utility of the training and/or 
technical assistance provided. To address intermediate outcomes evaluators also use surveys 
that ask about knowledge and perceived capability for helping districts and schools. Evaluators 
follow up on the effects of technical assistance projects by surveying participants about 
progress towards the long-term goals of applying new knowledge and improving services to 
districts and schools. 

Interviews and Focus Groups 

Interviews and focus groups are being conducted with key clients such as state-level contacts, 
NCCC technical assistance activity participants, and members of the NCCC Advisory Board to 
determine the continued quality and utility of the NCCC technical assistance. Interviews are 
also being used to evaluate the intermediate outcomes of increased knowledge and perceived 
capacity to help districts and schools. Evaluators are using follow-up interviews to determine 
how SEAs applied their knowledge and how they changed/improved their service to districts 
and schools (i.e., increased capacity). 
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Artifacts 

Relevant documents are an important source of evaluation data and are collected and reviewed 
on an ongoing basis. Artifacts documenting program inputs (e.g., activities and meetings) 
include meeting agendas and handouts. Artifacts supporting long-term outcomes are those 
that document the services to schools and districts provided by the SEAs and provide evidence 
of increased SEA capacity; these documents are expected to include legislation, state program 
plans and progress reports. Additionally, artifacts documenting tools and resources created by 
participants of technical assistance activities (book reviews, web sites) may also be used as 
evidence of capacity building. 

Analysis 

As mentioned above, NCCC technical assistance centers on four capacity building objectives: 

1. Increase state capacity to assess the improvement needs of districts and 
schools. 

2. Increase state capacity to develop solutions to address improvement needs 
of districts and schools. 

3. Increase state capacity to build and sustain systemic support for district and 
school improvement efforts related to closing achievement gaps and 
improving student outcomes 

4. Increase state capacity to improve the tools and systems for school 
improvement and accountability. 

These objectives were determined by the U.S. Department of Education and are used as the 
framework under which technical assistance is provided by all CCs. However, within this 
framework, the NCCC developed a set of indicators to measure progress for each objective (see 
Appendix A). Indicators were established in collaboration with NCCC state liaisons to reflect the 
outcomes of work under each project. Although each complete set of indicators reflects the 
scope of NCCC capacity building work under its corresponding objective, as project activities 
differ based on target skill development, different projects might encompass different sets of 
indicators. 

The data collection plan generates both qualitative and quantitative data and thus requires that 
a variety of appropriate analytic techniques be utilized. Data collected during individual and 
group interviews and document reviews primarily yield qualitative data. In general, qualitative 
data analyses use an analytic inductive strategy. Notes from interviews and focus groups are 
read and re-read, sets of tentative assertions are written, and general coding schemes 
containing preliminary variables are developed. As the analysis proceeds, searches for 
confirming and disconfirming evidence are undertaken and tentative assertions are revised as 
appropriate. Strategies such as member checking and triangulation are also undertaken to 
ensure the validity of the final sets of assertions. 
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Information about professional development events, activities and participant feedback 
collected by surveys generate quantitative data. Analyses of these data are largely descriptive, 
designed to document the nature of the work, the perceptions of stakeholders, and the 
perceived outcomes. Frequency distributions, cross-tabulations, measures of central tendency 
and dispersion, correlations and other statistics are used. Qualitative feedback from surveys 
and interviews are aggregated by content and reported as well. Qualitative feedback providing 
data on successful approaches and projects as well as suggestions for improvement is shared 
with NCCC staff to improve the technical assistance provided as appropriate. 

Results 

Most CCs span several states in a region and work on multiple projects in each state. While this 
CC will ultimately use the analysis strategy described above to evaluate the success of technical 
assistance activities for all states and projects in its region, we present findings here from Year 4 
for the state of Nebraska. 

During Year 4, NCCC services in Nebraska centered on the three main projects identified in the 
state management plan: the Balanced Leadership Professional Development Cadre, the English 
Language Learner Resources, and the Poverty Plan projects. Table 1 lists these projects, along 
with project status at the end of Year 4. Project activities are described in more detail in the 
following paragraphs. 



Table 1 

NCCC Year 4 Projects in Nebraska 



Nebraska Project 


Objective 


End of Year 4 Status 


Project 1: Balanced Leadership Professional 
Development Cadre 


3 


Project work completed 


Project 2: English Language Learner Resources 


3 


Ongoing in Year 5 


Project 3: Poverty Plan 


4 


Project work completed 



The Nebraska Balanced Leadership Professional Development Cadre is a continuation of work 
begun in Year 2. Since that time, NCCC has assisted the state with developing a cadre of 
professional developers who can provide training for administrators on research-based 
leadership practices associated with increased student achievement. In Year 4, this project 
focused on sustaining the ten active cadre members through quality assurance reviews and 
helping the cadre encourage administrators to participate in the training. 

The English Language Learner Resources Project is a continuation of work begun in Year 3. The 
goal of this project is to assist the SEA in developing resources and tools for educators working 
with the growing population of English language learners in Nebraska. SEA participants 
attended three training sessions in Year 3 to learn about effective instructional strategies for 
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English language learners. A design team was also established to help the SEA develop the 
resources and tools to disseminate to educators across the state. During Year 4, participants 
completed all courses in the Nebraska English Language Learner Leadership Institute training 
series and the design team continued to work on developing tools for teachers and the 
statewide framework. The Design Team held four working meetings during the reporting year, 
and continued to develop resources for educators that will be disseminated in Year 5. These 
resources become part of the state's online Continuous Improvement Process Toolkit, a process 
begun through an NCCC year 1 project. This work continues in Year 5. NCCC will continue to 
collaborate with the SEA to support the Design Team and will support the SEA in developing a 
training cadre to deliver professional development workshops based on Classroom Instruction 
that Works with English Language Learners to ESUs, districts, and schools. 

Under the Poverty Plan Project, the NCCC liaison assisted the Nebraska SEA to develop rubrics 
and tools to support districts in developing LEP and Poverty plans. The project provided a model 
for districts in developing comprehensive plans that address thirteen areas of best practices in 
supporting student achievement for LEP and high poverty students. The aim of this plan was to 
support SEA staff in analyzing the poverty data submitted by districts and interpreting the data 
relevant to state-level planning, and so the SEA can target technical assistance based on these 
plans. 

Documenting inputs and outputs. To evaluate Year 4 work, evaluators first documented 
project inputs and outputs, primarily using data from activity reports (Appendix B) and 
conversations with NCCC staff. While interesting for a comparative analysis of scope across 
projects, for a single project these data also allow researchers to calculate the cost per project 
in terms of staff hours, funding, and time. 

Documenting project outcomes. In addition, evaluators used multiple sources to document 
project outcomes. After each session, participants were asked to rate the quality, relevance, 
and utility of activities, as well as to respond to several open-ended questions. These exit 
surveys typically contained twelve Likert-type 1 items related to the GPRA Indicators on the 
specific value of the technical assistance and three open-ended items to allow participants to 
express additional thoughts about technical assistance activities. The three Likert items on the 

Table 1 




1 A response scale used as an indicator to differentiate between the most and least favorable items. 
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QA and support for Choosing 
the Right Focus session 


Jan. 14, 2009 


6 


22 


14 


41/42 

(98%) 


28/28 

(100%) 


28/28 

(100%) 


ENGLISH LANGUAGE LEARNER RESOURCES 


Design Team Meeting #3 


Sept. 19, 2008 


6 


14 


N/A 


mi 






Nebraska English Language 
Learner Institute (Omaha), 
session 4 


Oct. 1-2, 2008 


15 


52 


37 


lii/in 

(100%) 


74/74 

(100%) 


13/13 

(100%) 


Nebraska English Language 
Learner Institute (Kearney), 
session 4 


Oct. 27-28, 
2008 


15 


42 


41 


121/125 

(97%) 


81/84 

(96%) 


80/84 

(95%) 


Design Team Meeting #4 


Nov. 17-18, 
2008 


10 


12 


N/A 




m. 


m 


Nebraska English Language 
Learner Institute (Omaha), 
session 5 


Dec. 2-3, 2008 


12 


62 


37 


102/111 

(92%) 


74/74 

(100%) 


72/74 

(97%) 


Nebraska English Language 
Learner Institute (Kearney), 
session 4 


Dec. 4-5, 2008 


12 


47 


24 


66/72 

(92%) 


46/48 

(96%) 


47/48 

(98%) 


Design Team Meeting #6 


May 12, 2009 


6 


13 


N/A 


mz 


mz 


mz. 


POVERTY PLAN 


LEP focus and input group, 
meeting 1 


Feb. 19, 2009 


6 


20 


N/A 








LEP focus and input group, 
meeting 2 


April 23, 2009 


8 


29 


N/A 








Note: Numerators represent the number of 'excellent' or 'good' responses, 
denominators represent the total number of responses. 


441/46 

1 

(96%) 


303/308 

(98%) 


300/30 

7 

(98%) 



quality of technical assistance were on a four-point scale, from 'excellent' to 'poor.' The two 
Likert items on the relevance and the three items on the utility of activities were on a five-point 
scale, from 'to a great extent' to 'not at all.' NCCC established the target of 85 percent positive 
responses to survey items (responses of 'excellent' or 'good') to determine if activities were of 
high quality, relevance, and utility to project participants (see Table 2 for survey results). 

Aggregated exit survey results for Nebraska indicate that NCCC met its internal goals for the 
indicators of technical assistance activities: 96 percent of participants rated the quality of 
activities at excellent or very good, 98 percent of participants indicated the activities were 
relevant, and 98 percent of participants indicated activities would be useful in their work. Exit 
surveys for each project, and for each activity within each project, supported this positive 
trend. These indicator ratings on the short-term outcomes and impact of participation in NCCC 
technical assistance activities provided strong evidence that efforts made by the North Central 
Comprehensive Center team in building state capacity towards the projects identified in the 
Year 4 Management Plan are considered to be highly successful by project participants. 



8 




Documenting participant learning. Although responses on exit surveys indicated participants 
consistently perceived technical assistance activities to be of high quality, relevance, and utility, 
in line with the capacity building continuum evaluators next sought to determine more 
precisely what participants learned from project activities, and how they had applied what they 
learned. To this end, the year-end/end-of-project online surveys, interviews (Appendix C), and 
focus groups (Appendix D) captured the feedback of participants. These instruments, developed 
in collaboration with NCCC liaisons, were used to determine the level of satisfaction with the 
information and services provided under each project, as each state had a unique agenda that 
focused on its state's unique needs. Although input from many of these same participants was 
gathered via activity exit survey data, this summative measure provided them with the 
opportunity to reflect on the quality of NCCC assistance throughout the year with regard to a 
specific project. 

For the first Nebraska project, the Balanced Leadership Professional Development Cadre 
Project, evaluators developed an end of project survey using the indicators under objective 3 
related to this project (Appendix E; see results in Table 3). 

As the mission of the CCs is to build capacity at the state level, this survey was distributed to 
State Education Agency (SEA) representatives who were involved in the project rather than to 
all project participants. Quantitative data on participant application of knowledge gained from 
technical assistance was positive, indicating that participants had largely begun to apply 
knowledge gained from NCCC technical assistance, and was supported by the following 
comment: 

I have used the resources to support our principals in their leadership roles. The Balanced 

Leadership materials and training effectively translate theory into methods of practices - 

Nebraska Balanced Leadership Project 

Documenting self-efficacy and changes in practice. Evaluators invited participants from each 
site in the English Language Learner Resources project to participate in focus groups after the 
final training session. Results from these conversations are summarized in the intermediate and 
long-term outcomes columns of the logic model (see Figure 2). These data indicated that 
participants perceived an increase in their self-efficacy (intermediate goal) and were 
implementing changes in practice (long-term goal) in accordance with the goals of the project 
and the intent of the capacity building continuum. In addition, resources under development by 



Table 3 

Balanced Leadership Online Year-End Survey Results 




The technical assistance (e.g. information, resources, meeting facilitation...) provided by the NCCC 


staff related to the Project was: 

Cell numbers represent the fraction and percent of participants who responded 'excellent' or 'good' 


Research-based 


3/3 

(100%) 


Accurate 


3/3 
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(100%) 


Up-to-date 


3/3 

(100%) 


Related to my current needs 


3/3 

(100%) 


Practical 


3/3 

(100%) 


Applicable to my work 


3/3 

(100%) 


As a result of NCCC's technical assistance related to the Project, to what extent do vou 

perceive yourself as more capable of: 

Cell values represent the fraction and percent of participants who responded To a great extent' or 'to 

some extent' 


Understanding the components of the current support system 


3/3 

(100%) 


Reviewing existing statewide policies and revising procedures and practices 


Y//////////////A 


Developing measurable goals for implementation of the support system 


3/3 

(100%) 


Using the results of data analysis to guide decision making 


a//////////////z. 


Developing state capacity to communicate and collaborate, both externally 
and internally 


ill 

(100%) 


Leveraging existing resources to meet needs across the state 


3/3 

(100%) 


Providing differentiated support (ELL, PD, formative assessment, 
standards) to schools/districts 


111 

(100%) 


Establishing/maintaining statewide networks to enact improvement efforts 


3/3 

(100%) 



the Design Team presented evidence that participants were applying knowledge from project 
activities. 

Data collected from the Poverty Plan Project, completed in Year 4, also indicated that 
participants had increased capacity as a result of NCCC technical assistance activities. 

Evaluators conducted interviews with nine Poverty Plan participants. Participants were more 
capable and confident of developing a plan to support high poverty students as a result of NCCC 
technical assistance, as supported by the following responses: 

I found it useful in sharing ideas for each of the elements which must be 
addressed in a poverty plan, and the approaches and successes experienced by 
other districts. 

I am much more capable as the collaboration with other districts enhanced my 
knowledge of what is going on around the state and what components might be 
activities to be identified in the Poverty Plan. 

I was able to get a clearer vision of the difference between the document and the 
people part of the plan. The direct activities about which we will report are many 
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and I hope to build from where we are and expand considerably , while tapping 
into community resources. 

In additions, participants indicated they will use the resources and knowledge they gained 
through participation in the Poverty Plan session in some of the following ways: 

Develop a more detailed plan which will serve as a better guide rather than what 
we have had in the past and used more as a compliance document and filed 
away. 

Programming and or changes will be based as has been the case on data 
analysis. The information obtained from the session will be of assistance in 
looking at other potential interventions. 

Lessons Learned About Building Capacity 

Previous practice in evaluating the Comprehensive Center was focused primarily on 
documenting project activities and analyzing survey data on quality, relevance, and utility of 
project services. In contrast, the current evaluation of capacity building also focused on the 
cognitive and affective changes in participants (changes in knowledge and self-efficacy) and on 
behavior, such as changes in services to schools and districts. In considering the four 
overarching project objectives, evaluation findings showed participants not only reported that 
activities were useful, relevant, and of high quality - but that they perceived themselves as 
more knowledgeable and had begun to apply their knowledge towards improving services to 
schools and districts. 

Fully evaluating the capacity building continuum required more data than the more limited 
evaluations that had been conducted in the past. Therefore, more staff and participant time 
was needed in order to document the additional objectives. 

As part of the capacity-building focus of the evaluation, and in an effort to streamline the 
evaluation process, in Year 5 of the NCCC evaluation liaisons met with evaluators to identify 
capacity building indicators for each objective (Appendix E). Each set of indicators reflects the 
entire scope of NCCC work under the corresponding objective, as identified by NCCC liaisons. 
Relevant subsets of these indicators, identified by project liaisons, will be included in year-end 
and end-of project evaluation activities. 
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Figure 1. CC Logic Model— Evaluating Building SEA Capacity 



INPUTS 




OUTPUTS 




OUTCOMES -IMPACT 






^ Activities 


Participation 


^ Short term 


Intermediate 


Long term 


• Funding 

• CC staff 

• Technology 
(web site, etc.) 


1 


• Assist SEAs through 
state-specific 
projects 

• Annual Advisory 
Board meeting 

• Identify, broker, 
leverage, and 
disseminate 
information from 
Content Centers 


• Number of 
participants in each 
project activity 




• Participants will report 
that the project 
activities are useful, 
relevant, and of high 
quality 


• Participants will learn how 
to assist districts and 
schools, in relation to the 
targeted program 
objective, as a result of the 
project activities 

• Participants will perceive 
themselves to be more 
capable of assisting 
districts and schools, as a 
result of the project 
activities 


• Participants will apply 
the knowledge from 
the program activities 
to assist districts and 
schools according to 
the targeted program 
objective 

• Participants will 
improve services in 
one or more of the 
areas targeted by 
program objectives 


Evaluation Type and 
Questions Addressed 


Formative: How is the 
CC technical 
assistance being 
provided? (Further 
detail on what is 
collected from activity 
logs) 


Formative: How is 

the CC technical 
assistance being 
provided? (Who 
participates in CC 
activities? What are 
their roles?) 




Formative: How do the 
participants react to the 
technical assistance 
activities? What do they 
like and dislike about 
them? To what extent 
are the technical 
assistance activities 
found to be useful, 
relevant, and of high 
quality? 


Summative: What did 
participants learn as a result 
of the technical assistance 
activities, relative to the 
targeted objectives? To 
what extent do they 
perceive themselves as 
capable of assisting districts 
and schools, relative to the 
targeted objectives? 


Summative: How did 
the participants apply 
the knowledge from 
technical assistance 
activities in their 
services to districts and 
schools? What have 
they done differently? 
What evidence is there 
that services to districts 
and schools have 
improved? 


Data Sources 


Activity logs, activity 
reports, evaluation 
logs, artifacts 


Activity logs, activity 
reports, evaluation 
logs, artifacts 




Participant surveys 


Participant surveys, 
interviews 


Follow-up surveys and 
interviews, documents 
and artifacts 



ASSUMPTION: 

CC staff and SEA personnel will be able to work together to choose and implement program activities that address the four objectives 
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Figure 2. Results of English Language Learners (ELL) project aligned with logic model 



INPUTS 



• Approximately $5 
million in funding 
from the U.S. 
Department of 
Education, 
divided amongst 
projects in five 
states over five 
years 

• Planning and 
facilitation from 
one CC liaison; 
content delivery 
from three CC 
trainers 

• Approximately 
184 CC staff 
hours for Year 4 
work on this 
project 

• Balanced 
Leadership and 
Classroom 
Instruction that 
Works with 
English Language 
Learners, books 
and resources 



.at 



OUTPUTS 



Activities 



Participation 



English Language 
Learner Leadership 
Institute 

The goal of this activity 
was to help leadership 
teams develop capacity 
to improve instructional 
practices for English 
language learner 
students (ELLs) in the 
school setting. Activities 
centered on improving 
leadership and 
understanding research- 
based instructional 
strategies for effective 
instruction of ELLs. 

• Conduct session #4 

• Conduct session #5 
Design Team 

The goal of this activity 
was to develop a 
statewide framework 
and online resources for 
educators working with 
ELLs. 

• Facilitate design team 
meeting #3 

• Facilitate design team 
meeting #4 

• Facilitate design team 
meeting #6 



English Language 

Learner Leadership 

Institute 

• 40 participants at 
session #4 

• 50 participants at 
session #4 

• 45 participants at 
session #5 

• 60 participants at 
session #5 

Design Team 

• 14 participants at 
third design team 
meeting 

• 12 participants at 
fourth design team 
meeting 

• 11 participants at the 
sixth design team 
meeting 



Short term 



English Language Learner 
Leadership Institute 
(includes Design Team 
Participants) 

Averaged percent of 
participants who indicated 
project activities were of 
high quality, relevance, and 
utility in Kearney: 

Quality: 187/197 = 95% 
Relevance: 127/132 = 96% 
Utility: 127/132=96% 



Averaged responses on 
quality, relevance, and utility 
of project activities in 
Omaha: 

Quality: 213/222 = 96% 
Relevance: 148/148= 100% 
Utility: 145/147=99% 



Design Team 

Data on quality, relevance, 
and utility will be collected 
for these sessions once work 
towards the framework and 
online resources has been 
completed. 



OUTCOMES -IMPACT 
Intermediate 



English Language Learner 
Leadership Institute (includes 
Design Team) 

The Design Team is a subset 
of ELLLI participants. 
Responses from both groups 
suggested participants 
perceived an increase in self- 
efficacy, as indicated by the 
comments below. 

• BL helped me view my 
position to help others 
better and the security to 
say. I'll work with you on 
that. 

• I feel more effective as 
evaluator of ESL teachers. 
Original action plan seems 
naive; we know so much 
more now. 

• I learned a lot not included 
in my ELL endorsement 
classes. 

In addition, participants 
reported feeling: 

• More confident as a result 
of increased knowledge, 
confidence, and ability 

• Less burdened 

• More confident in using 
strategies for ELLS 

• More capable of supporting 
staff in understanding how 
ELLs learn and how to 
advance their achievement. 



Long term 



English Language Learner 
Leadership Institute (includes 
Design Team) 

ELLI participants reported 
changes in teacher practice 
including: 

• Using the resources and 
strategies in every lesson 
plan, lesson, lab and class 

• Working more with 
vocabulary 

• The feeling among staff 
that ELLs are part of our 
student population, not just 
the ELL teacher's problem. 

They also reported changes in 
their students including: 

• Increased vocabulary skills, 
reading skills, and test 
scores 

• A noticeable increase in 
self-confidence in ELL 
students. 

• ELL students are more 
empowered to speak in 
regular classes. 

Design Team 

• The CITW for ELL book 
study will be published 
online. 

• The Toolkit resources were 
revised and will be 
published 

• A video is under production 
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Appendix A 



Indicators of Capacity Building 

Objective 1: Increase state capacity to assess the improvement needs of districts and schools. 

a. Assess performance and practice in districts and schools to further improve performance 

b. Use a variety of data and systematic data collection methods to inform decisions 

c. Analyze and interpret data (e.g., student data, monitoring visits) 



Objective 2: Increase state capacity to develop solutions to address the improvement needs 
of schools and districts. 

a. Increased knowledge of the body of research 

b. Increased knowledge of effective research-based strategies 

c. Increased knowledge of the role of data to inform school improvement 

d. Increased knowledge of the role of stakeholders 

e. More capable of locating and using relevant resources 

f. More capable of providing expertise to your and/or other schools/districts 

g. More capable of identifying appropriate solutions (collaboratively, with schools/districts) 

h. More capable of providing support to schools/districts in implementing solutions 



Objective 3: Increase State capacity to build and sustain systemic support for district and 
school improvement efforts related to closing achievement gaps and improving student 
outcomes 

a. Understand the components of the current system 

b. Review existing policies and/or revising procedures and practices 

c. Define a system that is comprehensive and coherent, and integrates federal programs 
and/or state initiatives 

d. Use the research base to inform decision making 

e. Use the results of data analysis to guide decision making 

f. Develop measurable goals for implementation 

g. Use networks for internal and external communication 

h. Leverage existing resources 

i. Deliver high quality technical assistance, services and resources 

j. Provide differentiated support (e.g., ELL, SPED, Title 1, Indian students) to schools/districts 

k. Encourage leadership and collaboration 

l. Encourage the use of technology (e.g., professional development, data analysis, 
communication, monitoring) 

m. Scale up from implementation at pilot sites 



Objective 4: Increase State capacity to improve the tools and systems for school 
improvement and accountability 

a. Build awareness of existing tools/systems 
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b. Gather feedback/data on the usefulness of existing tools/systems 

c. Use feedback/data to determine what tools/systems are necessary 

d. Develop new tools/systems 

e. Disseminate new tools/systems 

f. Train schools and/or districts to use new tools/systems 

g. Monitor and evaluate implementation and/or impact of new tools/systems 

h. Implement a continuous improvement approach 
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Appendix B 



Activity Report 



Date of Request: 

Date of Activity (Date of Response): 

Total Hours: 

Level: State Regional 

Affiliation of Requester: Name of Activity: 

Nature of Activity (Please check one): 

( ) Increase State capacity to assess the improvement needs of districts and schools 

( ) Increase State capacity to develop solutions to address improvement needs of districts and schools 

( ) Increase State capacity to build and sustain systemic support for district and school improvement efforts related to closing 
achievement gaps and improving student outcomes 

( ) Increase State capacity to improve the tools and systems for school improvement and accountability 

Purpose/Objectives of the Activity: 



Activity Number: 
Location: 



Total Number of Participants with Names (NCCC staff and Key State Staff): 
Summary of the Activity (Response to Request): 



Details of the Activity (Things that happened during meeting): 



Follow Up/Next Steps: 



Lessons Learned about Increasing State Capacity: 



Did participants complete evaluations for this event? Yes No 

If yes, please specify which organization evaluated the activity and where the evaluation data can be 
found: 

Form Completed by: 
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Appendix C 



Nebraska Project 3: Poverty Plan 
Participant Interview Questions 



Objectives of NCCC: 

1. Increase State capacity to assess the improvement needs of districts and schools. 

2. Increase State capacity to develop solutions to address improvement needs of districts and schools. 

3. Increase State capacity to build and sustain systemic support for district and school improvement 
efforts related to closing achievement gaps and improving student outcomes 

4. Increase State capacity to improve the tools and systems for school improvement and accountability. 
Intermediate objectives from evaluation plan: 

1. What did participants learn as a result of the technical assistance activities, relative to the targeted 
objectives? 

2. To what extent do they perceive themselves as capable of assisting districts and schools, relative to 
the targeted objectives? 

Long term objectives from evaluation plan: 

1. How did the participants apply the knowledge from technical assistance activities in their services to 
districts and schools? 

2. What have they done differently? 

3. What evidence is there that services to districts and schools have improved? 

I'd like to explain the purpose of this interview. One objective of the NCCC is to increase state capacity 
to improve the tools and systems for school improvement and accountability. This evaluation 
component is to help determine whether and the degree to which this has been accomplished. 



1. What is your role in developing a poverty plan for your district? 

2. What do you see as the key activities in preparing the Poverty Plan? 

3. What was your reaction to the activities at the 23 April meeting? What did you like and dislike 
about them? 

4. What did you learn as a result of the Poverty Plan activities? 

5. Do you plan on using the rubric you developed in the near future? 

6. As a result of the 23 April meeting, to what extent do you perceive yourself as more capable of 
developing a plan to provide support to high poverty students? 

7. In what ways do you plan on providing this type of support to students in the near future? 

8. What will you do differently as a result of this activity? 

9. What additional activities, if any, would help develop your capacity to provide support to high 
poverty students? 
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Appendix D 



Nebraska Project: English Language Learner Leadership Institute 
Focus Group Questions 



Objectives of NCCC: 

5. Increase State capacity to assess the improvement needs of districts and schools. 

6. Increase State capacity to develop solutions to address improvement needs of districts and schools. 

7. Increase State capacity to build and sustain systemic support for district and school improvement 
efforts related to closing achievement gaps and improving student outcomes 

8. Increase State capacity to improve the tools and systems for school improvement and accountability. 
Intermediate objectives from evaluation plan: 

3. What did participants learn as a result of the technical assistance activities, relative to the targeted 
objectives? 

4. To what extent do they perceive themselves as capable of assisting districts and schools, relative to 
the targeted objectives? 

Long term objectives from evaluation plan: 

4. How did the participants apply the knowledge from technical assistance activities in their services to 
districts and schools? 

5. What have they done differently? 

6. What evidence is there that services to districts and schools have improved? 

I'd like to explain the purpose for this focus group. One component of the evaluation of the NCCC 
technical assistance is to determine whether and the degree to which the technical assistance 
provided has 'built the capacity' of state education agencies. 



1. What are your names and roles? 

2. What do you see as the key activities in the Nebraska English Language Learner Leadership Institute? 

3. What was your reaction to the activities? What do you like and dislike about them? 

4. What did you learn as a result of the NELLLI activities? 

5. How have you used the resources and information you have received through this project? (used by 
you alone, shared with others) In what ways do you plan on using these resources in the near (i.e., 
this school year) future? Design team: How has your work had an impact at the state level? 

6. In what ways does you ruse of the information different from others'? (urban, rural, etc.) 

7. To what extent do you perceive yourself as more capable of assisting districts and schools with ELLs? 
(For school level: assisting ELLs) 

8. In what ways do you plan on providing this type of support in the near (i.e., this school year) future? 

9. What have you done differently as a result of this activity? 

10. Describe for me the degree to which you believe teachers are making changes in their classroom 
practice. What evidence is there that services to districts and schools have improved? What 
evidence is there that relationships with students, parents, and communities have changed? 
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Appendix E 



NCCC Participant Evaluation Form 



NCCC Participant Evaluation Form 

NE Project 1: Balanced Leadership 



<fc»NE Project 1 : Balanced Leadership </b> <br/> 

Year Four Survey: July, 2008 - June, 2009 <br /> 

1 . 1 participated in the Balanced Leadership Project primarily in my role as a(n): (Required) 
O State Agency Staff 
O Intermediate Service Agency Staff 
O Regional Professional Development Provider 
O School Support Team Staff 
O School-Level Administrator 
O District-Level Administrator 

O Policy Maker 

O Other | 



2. The technical assistance (e g. information, resources, meeting facilitation...) provided by the 
NCCC staff related to the Balanced Leadership Project was: (Required) 





1 


1 

0 


LL 


1 


z 


a. research-based 


o 


o 


o 


o 


o 


b. accurate 


o 


o 


o 


o 


o 


c. up-to-date 


o 


o 


o 


o 


o 


d. related to my current needs 


o 


o 


o 


o 


o 


e. practical 


o 


o 


o 


o 


o 


f applicable to my work 


o 


o 


o 


0 


o 
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NCCC Participant Evaluation Form 



3. The objective of NCCC work under the Balanced Leadership Project was to increase state 
capacity to build and sustain systemic support for district and school improvement efforts relating to 
closing achievement gaps and improving student outcomes. As a result of NCCC's technical 
assistance related to the Balanced Leadership Project, to what extent do you percieve yourself as 
more capable of: (Required) 





To a Great 
Extent 


To Some 
Extent 


To a Small 
Extent 


To No 
Extent 


<: 

z 


a. understanding the components of the current support system 


o 


0 


o 


0 


o 


b. developing measurable goals for implementation of the support system 


o 


o 


o 


o 


o 


c. developing state capacity to communicate and collaborate, both externally and internally 


o 


0 


o 


0 


o 


d. leveraging existing resources to meet needs across the state 


o 


o 


o 


o 


o 


e. providing differentiated support (ELL, PD, formative assessment, standards) to schools/districts 


o 


o 


o 


o 


o 


f. establishing/maintaining statewide networks to enact improvement efforts 


o 


o 


o 


o 


o 



4. Please explain if/how NCCC technical assistance has increased your understanding of research- 
based leadership practices associated with increased student achievement. 




Page 2 of 3 
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NCCC Participant Evaluation Form 



7. What additional activities, if any, would help improve NCCC technical assistance in the Balanced 
Lea dership Pro ject to meet the needs of your schools/districts? 



Page 3 of 3 
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